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Abstract 

We briefly identify several critical issues in current computational neuroscience, and present our opinions on 
potential solutions based on bioimage informatics, especially automated image computing. 



Computational neuroscience is undergoing a transform- 
ation. Traditionally, this field has focused on studying 
information processing in nervous systems by collecting, 
analyzing, and simulating neuronal electrophysiology 
data [1,2]. Increasingly, the emphasis is shifting towards 
neuromorphological pattern analysis and brain atlas 
modeling using advanced microscopy and image com- 
puting. One important motivation for this changing 
focus is the need to ground our understating of animal 
behavior in real three-dimensional (3D) neuronal 
morphologies and connectivity. When dynamic imaging 
of living nervous tissue is possible, there is an additional 
opportunity to investigate the time-dependent molecular 
mechanisms in brain tissue in a data-driven manner. 
Advances in automated image computing are making 
such studies practical and informative. 

Image computing is critical for mapping brain 
and building neuron databases 

A common goal among many current studies is to map 
brain anatomy by systematically characterizing the distri- 
bution, projection, and connectivity of neurons through- 
out a nervous system. Pursuing this ambitious goal is 
increasingly practical, due to converging advances in 
neuron labeling [3,4], histological preparation [5], multi- 
dimensional imaging [6,7], and image computing [8,9]. 
Several large-scale 3D imaging-based computational neuro- 
science initiatives, such as the HHMI Janelia FlyLight 
(janelia.org/team-project/fly-light) and FlyEM projects 
(janelia.org/team-project/fly-em), the Allen Mouse Brain 
"MindScope" [10] and Connectivity Atlas (alleninstitute. 
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org/science/public_resources/atlases/connectivity_atlas.html), 
and the Harvard mouse brain connectome project (cbs. 
fas.harvard.edu/science/connectome-project), have been 
launched to acquire massive datasets, in which each image 
contains hundreds of millions of pixels. 

The task of storing, organizing, and disseminating 
these vast datasets has driven the development of several 
neuronal databases and digital brain atlases. For ex- 
ample, NeuroMorpho.Org is a web-accessible database 
that allows researchers to share neuron reconstructions, 
quantify neuronal morphology in a standardized manner, 
mine the resulting morphological features, and use them 
in computational modeling and neuronal network simu- 
lation studies [11]. The digital representations of light- 
level single neurons in NeuroMorpho.Org is associated 
with rich metadata, but only coarse information about 
the cellular location and orientation relative to sur- 
rounding brain structures. In contrast, 3D digital brain 
atlases contain rich and integrated information on the 
spatial coordinates of 3D reconstructed neurons, gene ex- 
pression, functional modules, and networks, all mapped 
into a standard' view of brain anatomy. Examples of re- 
cent digital atlases include a C. elegans connectome [12] 
produced from electron microscopic images, a large-scale 
single-neuron atlas (flycircuit.org [13]) and a stereotypy 
atlas of major neurite tracts [14] of the adult fruit fly brain, 
both produced from confocal microscopic images, a 
genome-wide gene expression map of the mouse brain 
produced using wide-field microscopic images (brain- 
map.org [15]), and a high-resolution reconstruction of the 
rat hippocampus [16]. 

Producing databases and atlases of neuronal structures 
from microscopy data entails a pipeline of image pro- 
cessing and analysis (Figure 1) (also see [9]). Key steps 
include (1) image pre-processing: correct images for 
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Figure 1 Major steps in building digital neuronal databases 
and brain atlases. After necessary image processing, the 3D 
morphology of a neuronal pattern is reconstructed from microscopy 
data. Multiple neuronal patterns need to be organized based on brain 
regions and neuronal types. For brain atlases, these reconstructions are 
mapped into a common 3D spatial coordinate system. This may 
involve an iterative analysis process. 



imaging artifacts and standardize the raw data; (2) regis- 
tration: align brain morphology and stitch together sep- 
arately imaged brain regions; (3) segmentation and 
validation: trace neuron morphology and ensure that the 
reconstructions are correct; (4) feature extraction: make 
detailed quantitative measurements of traced neurons; 
and (5) analytics: model, infer, and predict the identities 
of neurons or neuronal patterns using morphology, loca- 
tion, genetic, lineage, and other extracted features. Many 
of these steps, such as 3D neuron tracing, are also widely 
used in smaller-scale analysis of neuronal phenotypes. 

The challenge of full automation 

The long-standing need to automate the laborious and 
subjective manual analysis of light-microscopic and 
electron-microscopic images has motivated a large num- 
ber of bioimage informatics efforts [8,17-21]. The recent 
spurt in imaging throughput, combined with the desire 
for large-scale computational modeling, has added a 
sense of urgency to this need. For instance, manual tra- 
cing of neuron morphology is prohibitively expensive for 
analyzing image data that is approaching the scale of 
terabytes and thousands of image stacks, let alone min- 
ing the higher-order associations in these data. 

For computational neuroscience, this great demand 
motivated several major institutions to conduct the 
worldwide DIADEM Challenge (diademchallenge.org) in 
2010 as a way to stimulate progress and attract new 
computational researchers to join the technology devel- 
opment community. The challenge was to develop 



algorithms capable of automatically tracing stacks of im- 
ages visualizing the tree-like shape of neuronal axons 
and dendrites into 3D digital reconstructions. This com- 
petition succeeded in stimulating a burst of progress. 
Five finalist-teams were selected from more than one 
hundred entries. A novel metric methodology was cus- 
tom designed to compare automated and manual recon- 
structions [22] . While none of the algorithms presented at 
the finishing stage reached the official goal of a 20-fold 
speed-up in the reconstruction process, all finalists' soft- 
ware programs were made publicly downloadable along 
with the challenge data sets and the DIADEM metric, 
providing a useful toolbox for continuous and future 
development. 

To achieve full automation, libraries of computer pro- 
grams for neuronal pattern extraction, comparison and 
inference must be integrated into toolkits for multi- 
dimensional image visualization and analysis (e.g. vaa3d. 
org [23]; farsight-toolkit.org). Optimizing each step of 
automated image computing (Figure 1) is also important 
to ensure a lower error rate compared to humans. In 
addition to a metric of precision, automation of image 
computing should also be assessed using measures of 
the speed-up compared to manual work, as well as the 
robustness of the method with respect to changes of its 
parameters and the type and level of noise in the data. 
These aspects relate to the scalability to large-scale ap- 
plications and the generalization to similar or new prob- 
lems. The above considerations apply to image data 
produced by most imaging modalities. Despite consider- 
able progress, many computational problems remain 
open. For instance, the DIADEM neuron reconstruction 
challenge stands unmet. Part of the reason could be that 
only relatively small image data sets on neuron morph- 
ology were previously available to researchers. We hope 
this situation will change soon due to the availability of 
benchmark datasets and the strong demand of very 
large-scale single neuron screening projects in the next 
five years. 

Making computer programs more intelligent 

Increasingly, researchers have developed ways to make 
some of these algorithms more intelligent. Domain- 
specific prior knowledge can be used to improve the 
"intelligence" of an algorithm, akin to the observation 
that a "supervised" machine-learning algorithm generally 
performs better (e.g. predicts more accurately) than an 
"unsupervised" one. How to incorporate domain know- 
ledge in an algorithm? One way is to code this informa- 
tion into explicit rules. For instance, the spatial location 
of cells coded as directed graphs has been used as con- 
straints in training a nuclear image segmentation and 
recognition algorithm [24]. In another example, particu- 
lar types of neuron tracing errors could be propagated 
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through a large set of results to identify other potential 
erroneous reconstruction loci [25]. However, in many 
cases it is hard to enumerate all possible rules and code 
them explicitly. Rules may also become contradictory, 
biased, or "fuzzy" under different circumstances. There- 
fore, another more competent way to provide the do- 
main knowledge is needed. 

Interesting solutions may originate from comprehen- 
sive neuron databases and brain anatomy atlases. The 
reciprocal relationship that links neuroscience databases 
and atlases with image computing methods offers a syn- 
ergistic opportunity to improve both of them iteratively. 
In both atlases and databases, it is possible to quantify 
neuronal patterns from many samples statistically. The 
statistics provide good "rules" or "priors", and new para- 
digms, for improving image-computing algorithms. For 
example, since an atlas of cells is a comprehensive model 
of the expected yet spatially deformed objects (cells) in 
the observed image data, a model-based search approach 
can be designed to achieve registration, segmentation, 
and recognition at the same time. An automated algo- 
rithm for simultaneous cell segmentation and recogni- 
tion of C. elegans cells [26] uses a standard 3D nuclear 
atlas of this animal to achieve much better performance 
than the more intuitive approach of performing these 
tasks in separate steps. 

When neuron locations are stereotypical, atlas-based 
image- computing methods can be powerful to design 
novel biological experiments. The spatial variation of 
most nuclei, including many neurons, in freshly hatched 
C. elegans is about 2 microns [24], just about the average 
size of a nucleus. For adult fruit fly brains, this variation 
for the major neurite tracts is merely ~3 microns, orders 
of magnitude smaller than the dimensions of an entire 
brain [14]. Therefore, it is possible to design new 
computer-driven instruments to repeatedly target the 
same neuron or neurite tract, locally perturb the neur- 
onal circuits of these animals in real time experiments, 
and observe the response of the nervous system. Among 
ongoing efforts, the Janelia SmartScope project aims at 
leveraging new image-guided computer programs to 
drive a 3D laser-scanning-microscope to target and 
stimulate single cells precisely in 3D and observe their 
downstream activities, using optogenetics and calcium 
imaging. 

Studying dynamics and functions of neural circuits 

Functional connectivity and dynamics of neuronal con- 
nections are two other important topics of computa- 
tional neuroscience. Neuronal dynamics are reflected in 
structure and activity at a variety of temporal scales, ran- 
ging from arbor growth and cell migration (days to years), 
through dendritic spine twitching and axonal bouton 
crawling (minutes to hours), to neuron firing (milliseconds 



to seconds). Intravital imaging [27] provides excellent 
means to visualize these temporal changes. The dynamics 
can then be studied using four-dimensional computational 
image analysis. For instance, tracking the 3D morpho- 
logical change of neurons over time from time-lapse im- 
aging data offers great opportunity to improve neuronal 
classification and characterize the structural plasticity of 
growing, degenerating, or regenerating neurons [20]. 
Anchoring corresponding branching points of varying 
neuron morphology and detecting persistent and variable 
structures (branch tips, boutons, spines) from multi- 
dimensional image series are useful steps to analyze neur- 
onal dynamics. Although the automation of these image 
analysis tasks is still new, it is highly desired to quantita- 
tively study these problems at large scale. 

In another example, 3D segmentation and tracking of 
somatic calcium imaging data provide a powerful way to 
study neuron activity patterns. Computational causal 
analysis might also enable interesting analysis of synaptic 
connections [28] (see however [29]). Imaging neuronal 
firing with possibly genetically encoded voltage -gated 
dyes or fast calcium indicators could enable the co- 
detection of neural network structure and activity. Using 
brain atlases and computational image-guided ana- 
lysis, it is possible to pinpoint the firing sites in vivo 
and record the firing patterns in real-time experi- 
ments. This approach could complement the recent 
non-image-guided automatic whole-cell patch-clamp 
electrophysiology [30]. 

Correlative analysis of microscopic image data of neu- 
rons and other functional data, such as electrophysiology 
and optical-physiology recording and behavioral assays, 
can become a powerful approach to gain useful informa- 
tion and knowledge from a large amount of data, and 
may lead to the ultimate understanding of how individ- 
ual neurons and their networks work. For instance, 
digitization of the dragonfly descending neurons' morph- 
ology from confocal images has been correlated with the 
electrical recording and measurement of the receptive 
fields of these neurons [31]. This analysis led to a new 
finding that the average population vector of the neur- 
onal response of a surprisingly small number of de- 
scending neurons actually precisely controls the wings of 
the animal in targeting preys [31]. On the other hand, 
for behavior assays, systematic video-analysis based 
high-throughput studies have yielded powerful new ways 
to associate the neuronal circuits of both insects and 
mammals with either the stimuli (such as the mouse 
whisker system [32]) or the behavior output (such as the 
Drosophila trajectories [33]). In the work of [33], a ma- 
chine learning system enables biologists to automatically 
annotate the behaviors of animals in each frame of a 
video. It can be used to detect a wide variety of social 
and locomotion behaviors across flies, mice, and larvae. 
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Recent years have witnessed an acceleration of the on- 
going transformation of computational neuroscience due 
to image computing automation. For example, neurite 
outgrowth assays employing automated reconstruction 
of neurons in high-throughput high-content environ- 
ments are now common, and are used for testing hy- 
potheses, screening, and toxicology studies [34-36]. At a 
more advanced level, new high-throughput methods for 
accurate neurite reconstruction [37] have allowed ana- 
lysis of the specific wiring underlying computation of 
direction-selectivity in retinal circuits [38]. More gener- 
ally, these kinds of structural neurobiology studies may 
systematically provide the much needed mechanistic un- 
derstanding of neural computation [39] if employed in 
large-scale automated applications [40] of new tech- 
niques for the anatomical and molecular interrogation of 
intact biological systems [41]. 

Future prospects 

We are optimistic that in the next five years more auto- 
mated and intelligent computer programs for analyzing 
large-scale brain images will yield tangible advancements 
in computational neuroscience. The neuron anatomy 
projects for various model animals will start to deliver 
integrated data sets, organized in neuron databases and 
brain atlases. This will provide invaluable data resources 
for the research community to analyze. This may lead to 
a paradigm shift similar to the massive rise of mining 
and reanalysis of secondary' (archived) data following 
the sudden abundance of genome sequences 10 years 
ago. Neuron databases and brain atlases are also going 
to incorporate more dynamic and functional informa- 
tion, such as from time-lapse and calcium imaging 
of interacting neurons. More realistic computational 
models of neurons and their networks may foster a 
deeper understanding of the functioning mechanisms of 
nervous systems, paving the path to re-create a brain in 
silico. 
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